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BACKGROUND OF INVENTION 

[0001] The invention is directed to a method for automatically determining a 

foreground color for a digital image, in particular, for placing a text or graphic object 
against a background image. 

[0002] In many different areas, a text or a graphic object is combined with an 

image to be displayed together. In these cases, usually the text or the graphic object 
constitutes a foreground object whereas the image is the background. A combination 
of the text and image is used, for example, for esthetical reasons or to illustrate the 
text by means of the background image. 

[0003] A graphic designer is often confronted with the task of adding a 

foreground text object to a given background image in such a way that the text is 
clearly legible. In such a case, the selection of a suitable color poses a problem due to 
the usually large number of possible and available colors for the text object. 

[0004] Furthermore, a combination of a text object with a background image 

is encountered on many web pages where the text is provided on a colored 
background. However, web pages are to be displayed on a variety of different devices 
such as different types of computer monitors or of handhelds or mobile phones. 
Depending on the capabilities of the device or the graphic card, the text and the 
background image are displayed differently. In particular, a designer cannot exclude 
that in some cases, the text and the background image are displayed in such a way that 
the color of the text and the color or colors of the background image are almost the 
same which makes it difficult to read the text. 

[0005] Methods for selecting a color are known from different fields. For 

example, U.S. Patent No. 5,508,718 discloses a computerized color selection system 
which selects colors based on a user's designation of color impact objectives which 
specify how colors affect those who view the colors rather than specifying the colors 
themselves. Other methods for color selection or for evaluating graphical data are 
known, for example, from U.S. Patent Nos. 5,889,932, 5,424,945, 5,909,220, and 
5,615,320, and U.S. Patent Publication 2001/0049591 Al. 
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[0006] Notwithstanding these advances in color selection, there continues to 

exist a need for automatically determining a foreground color for a given background 
image. It would also be desirable that such a method for automatically determining a 
foreground color for a digital image allows an improved representation of a text or 
graphic object against an image background. 

SUMMARY OF INVENTION 

[0007] In accordance with the invention, there is provided an apparatus, 

method and article of manufacture therefor, for automatically determining a 
foreground color for a digital image. The apparatus includes a color clustering module 
and a color selection module. The color clustering module automatically divides the 
colors of the pixels of at least a part of the digital image into a number of color 
clusters in a color space. The color selection module automatically selects for at least 
one color cluster, a color that is related to the at least one color cluster according to 
predetermined criteria. Examples of digital images include pictures and photographs, 
as well as colored background images. 

[0008] In accordance with one aspect of the invention, the color clustering 

module is adapted to analyze a digital image in color spaces, such as RGB, HSL, or 
L*a*b*, to identify colors that dominate the digital image. In accordance with another 
aspect of the invention, the color selection module provides a "candidate" color or 
recommended color depending on the dominant colors identified by the color 
clustering module. 

[0009] In accordance with yet another aspect of the invention, the color 

selected by the color selection module, for the at least one color cluster, has certain 
relationships to specific predetermined criteria in that it may be "close" in some 
respects and "different" in other respects. The predetermined criteria are responsible 
for the way the selected color differs from the color cluster. For example, the criteria 
can be based on a predetermined legibility criterion of the selected color with respect 
to the cluster in color space. 

[0010] Advantageously, the method may be adapted to bring about a well 

designed (i.e., esthetically pleasing), legible document, for example, with a 
foreground text or graphics object in the selected color. Thus, in particular, for a 
specific display device, the method may be adapted to automatically determine a 
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suitable foreground color for a graphics or text object depending on how a foreground 
object and a digital background image would be displayed on the device. 

[0011] According to various embodiments of the invention, the color 

clustering module converts image data of the digital image to a predetermined color 
format. The image data are the color values of the pixels of the digital image. In this 
way, the color format or the color space can be chosen in a suitable way depending on 
the further processing of the data, in particular, depending on the predetermined 
criteria for selecting a color. 

[0012] According to various embodiments of the invention, the color 

clustering module divides the colors of the pixels into clusters in a color space in one 
of a plurality of ways that includes using an Expectation-Maximization clustering, 
which yields an advantageous implementation for a predetermined number of clusters. 

[0013] According to various embodiments of the invention, the color 

clustering module determines the number of clusters using a model selection method. 
In particular, the model selection methods can comprise determining the number of 
clusters using a Bayesian Information Criterion or a Universal Model-based Minimum 
Description Length Principle. In this way, the number of clusters into which the 
colors of the pixels are divided need not be known in advance and can be determined 
for each case separately. However, it is also possible that a user determines the 
number of clusters or alternatively the number of clusters may be a predefined fixed 
number of clusters. 

[0014] In accordance with yet a further aspect of the invention, color selection 

may be preceded by image segmentation, which involves automatically segmenting 
part of the digital image into regions according to the color clusters. Advantageously 
image segmentation (performed by an image segmentation module) provides that the 
areas of the digital image are assigned cluster labels corresponding to the color 
clusters in order to determine the local colors of the image. 

[0015] According to various embodiments of the invention, image 

segmentation is performed using a normalized cut criterion or an energy-minimization 
method. In this way, image segmentation can be performed using a method that is 
adapted for the images to be processed and/or the processing device. 
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[0016] 



According to various embodiments of the invention, the image 



segmentation module groups the color clusters output by the color clustering module 
into interference clusters comprising a pixel in a selected region and benign clusters 
comprising no pixel in the selected region, and selecting a color being related to all 
interference clusters according to predetermined criteria. Advantageously, image 
segmentation provides that when the color selection module automatically selects a 
color, all clusters are taken into account which have regions that overlap with the 
selected region (where the foreground text or graphic object is to be placed). 

[0017] According to various embodiments of the invention, a color can be 

selected by the color selection module according to a legibility criterion for a 
predetermined foreground region. A legibility criterion indicates how well a text is 
legible. This enables the placement of a foreground region having improved legibility 
against a given background image. 

[0018] According to various embodiments of the invention, a color can be 

selected based on a likelihood ratio of the hypothesis that the digital image contains 
the foreground region and the hypothesis that the digital image does not contain the 
foreground region. In these embodiments, the legibility criterion is based on the score 
for these two models that the image contains text and that the image just contains 
background. 

[0019] According to various embodiments of the invention, selecting a color 

according to a legibility criterion includes computing local measures of contrast 
between background and foreground in a neighborhood of a candidate region of the 
background image, and computing a value that is representative of the lowest few 
contrast values observed over the candidate region. In one embodiment, the legibility 
score can comprise maximizing the legibility min max r{x + y) , wherein 



denotes heuristic or other models of likelihoods that the image / contains text T at a 
given pixel x and Pr(/(jc) | B) denotes heuristic or other models of likelihoods that 

the image / contains background B at the given pixel x . 



r( x ) = log 



Pr(/(x) | T) 
Pr(/(x) | B) 



, C 2 E is a disc of radius £ , and wherein Pr(/(x) | T) 
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[0020] According to various embodiments of the invention, a color can be 

selected according to a color harmony criterion. Selecting a color according to a color 
harmony criterion can be performed alternatively or additionally to a legibility 
criterion. 

[0021] In particular, a color can be selected according to at least one of a 

monotonic, a complementary, and a p-adic color harmony criterion in HSL space. A 
monotonic color harmony is present if there is a small distance (approximately 0°) 
between two colors in hue. A complementary hue is given by two colors having a hue 
which is about 180° apart in HSL space. By dividing the hue in HSL space into p 
equal or nearly equal increments over a range of 360°, a p-adic (e.g., triadic, quadric, 
etc.) color harmony is obtained. According to a further embodiment, a color can be 
selected according to a color harmony criterion with respect to at least one 
interference cluster and/or one benign cluster. 

[0022] According to various embodiments of the invention, the color selection 

module determines a color subset according to a color harmony criterion and 
maximizes a legibility function in the color subset. In this way, a color harmony 
criterion and a legibility criterion are combined. Advantageously, determining a 
subset based on a color harmony criterion reduces the number of possible colors for 
which the legibility criterion is maximized. In particular, determining a color subset 
can comprise sub-sampling the color subset. 

[0023] According to various embodiments of the invention, a color c can be 

selected using a linear design function for which 

M N 

Z*^)+Zr**(c.^) 

z=l k=\ 

is maximal, wherein P. denote the interference clusters, K k denote all clusters, both 

benign and interference, / is a legibility function in color space, h is a color harmony 
function, and a i and y k are weighting factors. 

[0024] The legibility function can have different forms; for example, it can 

represent the standard Euclidean distance in color space. As an alternative, the 
luminance coordinate can be weighted higher than the other coordinates. According to 
a further embodiment, the legibility function can correspond to a legibility criterion as 
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described above. The design function can represent a color harmony criterion as 
mentioned above or other design principles, for example, such that the distance of the 
resulting color to one of the benign clusters is small. 

[0025] According to various embodiments of the invention, the above- 

described methods can further comprise displaying or storing a predetermined graphic 
or text object using the selected color together with the digital image. In this way, a 
combined image is obtained, wherein the graphic object or text object in the 
foreground has a color being optimized with respect to legibility against the 
background image. 

[0026] In accordance with a further aspect of the invention, there is provided a 

method for determining legibility of an image having an identified foreground and 
background. The method includes computing local measures of contrast between 
background and foreground in a neighborhood for a predetermined foreground region 

of the image, and computing a legibility score representative of a lowest few contrast 

\ 

values observed over the predetermined foreground region. 

BRIEF DESCRIPTION OF DRAWINGS 

[0027] These and other aspects of the invention will become apparent from the 

following description read in conjunction with the accompanying drawings wherein 
the same reference numerals have been applied to like parts, and in which: 

[0028] Figure 1 illustrates elements of an example system for automatically 

determining a foreground color according to the invention; 

[0029] Figure 2 is a flow diagram illustrating an example of a method for 

automatically determining a foreground color according to the invention; 

[0030] Figures 3 illustrates an example background image; 

[0031] Figure 4 illustrates an example of clustering the pixels of the 

background image shown in Figure 3; 

[0032] Figures 5 and illustrates an example results of segmenting the image 

shown in Figure 3; 

[0033] Figure 7 illustrates interference and benign color clusters for a 

foreground block 704 that intersects a background image segment; 
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[0034] Figure 8 illustrates an example of the background image shown in 

Figure 3 displayed together with foreground regions; and 

[0035] Figure 9 is a flow diagram illustrating a method for recommending a 

color for foreground of a background region of an image. 

DETAILED DESCRIPTION 

[0036] A. Methods For Determining Foreground Color 

[0037] Figure 1 illustrates elements of an example of a system 100 for 

performing a method for automatically determining a foreground color. The system 
100 comprises a color clustering module 101, a (optional) color segmentation module 
102, and a color selection module 103. A digital image 111 is input to the color 
clustering module 101 and the color segmentation module 102. This digital image 1 1 1 
is intended as the background image. It is the aim to present a (predetermined) 
foreground text or graphic object against this background image in a selected region 
of the background image. The foreground color to be determined for the foreground 
text or graphics object depends on the region of the image where the text or graphic is 
to be placed. In view of this, a selected region (also called a candidate region) 1 13 is 
to be provided. 

[0038] In the example shown in Figure 1, this candidate region 113 is 

provided to the color selection module 103. The region 113 can be provided by a user 
via a suitable interface. For example, the digital image 1 1 1 can be displayed on a 
monitor and a user can select a region 113 using a cursor or mouse pointer. 
Alternatively, the candidate region 113 can be selected automatically using 
appropriate image processing methods with appropriate criteria. 

[0039] Instead of the case illustrated in Figure 1, it is also possible to provide a 

pre-selected candidate region 113 to the color clustering module 101. This is 
particularly useful if color clustering, color segmentation and color selection are not 
performed for the whole image but only for the part of the image, for example, 
consisting of the selected region or a larger area comprising the selected region. 

[0040] Upon completing color selection for the digital image, the 

recommended color or colors output at 115 for the foreground text or graphics object 
input at 1 1 7 intended to be placed on the background image 111 may be combined or 
overlaid by image coalescing module 118 and output (or simply provided as a 
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recommendation) at 119. It is to be understood that more than one text or graphic 
object can be placed against the background of a digital image 111. In such a case, 
more than one candidate region 113 is to be provided resulting in a recommended 
color for each candidate region. 

[0041] It will be appreciated that the method according to the invention and 

the exemplary system shown in Figure 1 can be used in various environments. In 
particular, it can be used for recommending different foreground colors when 
presenting a web page on different display devices. Many web pages are designed 
with elements where a text is placed over a photograph or another colored 
background. Doing such design automatically without the benefit of foreground 
recommendations provided by the invention, creates a risk that a color and/or a 
placement would be chosen such that the text is not legible or that colors would be 
chosen that do not coordinate with the overall color scheme of the image. 

[0042] Such a color foreground selection problem particularly arises if a 

corresponding web page is displayed on a device having a different color resolution or 
a different display area than expected by the web designer (for example when 
displaying on a mobile phone or a personal digital assistant (PDA)). In such 
situations, the web page is actually displayed with different colors or with the image 
scaled to a certain dimension. In the event, the web designer saves the text or graphic 
object together with the background image in a single bitmap file, a scaling or 
displaying with different colors (depending on the display device) could render the 
page illegible. Thus depending on expected display conditions, it would be 
advantageous to provide the background image and the foreground object separately 
such that an appropriate color for the foreground image can be determined that is 
adapted to the display conditions using the method according to the invention. Such 
an adaption, including providing a recommended color, could be performed on the 
web server (after having received the display characteristics of the display device) or 
by the display device (such as a mobile phone or a PDA) itself. 

[0043] Another environment in which the present invention may be 

advantageously used is within the context of automated document layout (ADL). By 
providing an ADL system with the ability to combine images with graphics and text 
in an appropriate way increases the range and the quality of documents that can be 
produced. In one embodiment, such an ADL program which automatically places text 
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or graphics on images can be implemented within a graphic art editing program that 
provides foreground color recommendations in accordance with the invention. 

[0044] Furthermore, a color matching method and system 100 as described 

herein can be used as a design aid for modifying an existing environment with a new 
element. For instance, the system 100 may be adapted to provide recommendations 
for a user that wishes to add a sign to a storefront or choose the color of a rug to 
complement existing furniture. In such a case, the user would take a photograph of the 
scene, upload it into the computer, analyze the image using the method according to 
the invention and even see the resulting image modified with the new element. 

[0045] A flow diagram illustrating an example of a method according to the 

invention is shown in Figure 2. Initially, a digital image 111 is provided to the 
corresponding system 100. 

[0046] At step 201, the image data (the color values of the image pixels) is 

converted to a predetermined color format. Many color formats or color spaces are 
possible, for example, RGB, HSL, L*a*b*, etc.). RGB is a common color 
representation for computers and HSL is commonly used in graphic arts. Another 
alternative is the L*a*b* which is based on the physiology of human color perception 
where the distance corresponds to human color discrimination. Depending on the 
intended use of the foreground object with the background image, different color 
spaces can be appropriate. 

[0047] At step 202, the colors of the formatted image data are clustered into a 

small number of groups or labels in order to determine the dominant colors of the 
overall document (digital image). In many cases, for example, if the background 
image is a photograph, the image may contain many colors related in a complex way. 
Such a phenomenon can arise for several reasons. Textures often contain many colors. 
Shading means that an object of a single hue contains many colors of differing 
luminance. Colored light sources and reflections from nearby objects can result in a 
variation in hue of the colors of an object. Image degradations such as noise and blur 
can also create additional colors. 

[0048] In the field of pattern recognition, different methods for clustering are 

known. An advantageous family of methods is based on the Expectation- 
Maximization (EM) method as applied to estimating a Gaussian mixture model. In 



this method, one starts with a given number of clusters. A corresponding number of 
data points are taken randomly as cluster means (centroid, center of gravity). 

[0049] In the EM expectation step, for all data points, the conditional 

probability is determined that a data point arises from each cluster using a MAP 
(Maximum A Posteriori) principle. In the EM maximization step, the parameters of 
the clusters are re-estimated given the conditional probabilities of the data. These 
steps are repeated until the overall likelihood changes less than a predetermined 
threshold or after a predetermined number of iterations. 

[0050] In the embodiment described above, clustering at step 202 is performed 

for a given number of clusters. However, usually this number is not known in 
advance. In view of this, step 202 can be repeated several times, each time with a 
different number of clusters, for example, starting with one or two clusters and 
repeating the step up to a predetermined maximum number of clusters. 

[0051] At step 203, the optimal clustering is determined. In this step, the best 

number of clusters can be determined by several model selection methods including a 
Bayesian Information Criterion or a Universal Model-based Minimum Description 
Length Principle. Alternatively or additionally, other criteria such as the Integrated 
Completed Likelihood (ICL), the Normalized Entropy Criterion (NEC), or the cross- 
validation criterion (CV) can be used for model selection. In one alternate 
embodiment at 203, a user may determine the optimal number of clusters. In yet 
another embodiment at 203, the optimal number of clusters may be a predefined fixed 
number of clusters. 

[0052] An example of Expectation-Maximization clustering for one to four 

clusters is shown in Figure 4 for an example image shown in Figure 3. For this 
example of Expectation-Maximization clustering, the example image shown in Figure 
3 represents a background image comprised of a red fox on a blue background. More 
specifically, Figure 4 shows the data points in color space (RGB) with the results of 
Expectation-Maximization clustering for one cluster at 401, two clusters at 402, three 
clusters at 403, and four clusters at 404. The ellipsoids shown in Figure 4 are the 50% 
probability ellipsoids of the corresponding Gaussian distribution. 

[0053] Continuing with step 204, image segmentation is performed. In this 

step, each area of the whole image or a part of the image (depending on whether the 
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color clustering was performed for the whole image or only a part of it) is assigned a 
cluster label in order to determine the local colors of the image 111. In particular, the 
goal at 204 is to label each pixel such that the fit of the pixel to the label is high and 
the piecewise smoothness of the region is preserved. Different methods are possible 
for carrying out image segmentation at 204. 

[0054] A first embodiment for performing image segmentation is based on a 

normalized cut criterion. This method is described, for example, in J. Shi and J. 
Malik, "Normalized Cuts and Image Segmentation", IEEE Transactions on Pattern 
Analysis and Machine Intelligence 22, 888- 905 (2000), which is incorporated herein 
by reference. Further alternatives are region-growing, region-splitting, or energy- 
minimization methods. Additionally, other methods of image segmentation known in 
the art may be used as will be appreciated by those skilled in the art, some of which 
are disclosed in the following publications, which are all incorporated herein by 
reference: K.S. Fu and J.K. Mui, M A Survey On Image Segmentation", Pattern 
Recognition, vol. 13, pp. 3-16, 1981; R.M. Haralick, "Image Segmentation Survey", 
in Fundamentals in Computer Vision, O. D. Faugeras (ed.), pp. 209-224, Cambridge 
University Press, Cambridge, 1983; R.M. Haralick and L.G. Shapiro, "Survey - Image 
Segmentation Techniques", Computer Vision Graphics and Image Processing, vol. 29, 
pp. 100-132, 1985; N.R. Pal and S.K. Pal, "A Review On Image Segmentation 
Techniques", Pattern Recognition, vol. 26, pp. 1277-1294, 1993; and P.K. Sahoo, S. 
Soltani, A.K.C. Wong and Y.C. Chen, "A Survey Of Thresholding Techniques", 
Computer Vision, Graphics and Image Processing, vol. 41, pp. 233-260, 1988. 

[0055] As an example, image segmentation was performed for the image 

shown in Figure 3. For this image, Expectation-Maximization clustering with three 
clusters at 403 (as shown in Figure 4) yields the best result according to model 
selection. Image segmentation was performed using the best result according to model 
selection. Thus, image segmentation was performed using a graph cut method and the 
three cluster Expectation-Maximization model. The result of this segmentation is 
shown in Figures 5 and 6, in which Figure 5 shows a first segmented portion of the 
image in Figure 3 (which is made up of one of the three clusters at 403) and Figure 6 
shows a second segmented portion of the image in Figure 3 (which is made up of two 
of the three clusters at 403). 



[0056] Returning to Figure 2, the method continues with the cluster grouping 

of step 205. In order to perform this step 205, a selected region or candidate region is 
to be provided where a foreground object is to be placed. This region should be large 
enough to contain the foreground object and possibly include an amount of padding 
around the foreground object. Then at 205, the color clusters are divided into two sets: 
the clusters which have regions that overlap with the foreground region (i.e., 
interference clusters) and the clusters which do not have regions overlapping with the 
foreground region (i.e., benign clusters). It may happen that all clusters are 
interference clusters. 

[0057] In alternate embodiments, the grouping of clusters performed at 205 

may be performed without first segmenting the image at step 204. Thus in an alternate 
embodiment, colors may be grouped at 205, for example, by labeling pixels with a 
most likely cluster and finding the most frequently occurring label. In another 
embodiment, grouping is optional at 202, for example, by considering all clusters as 
interference clusters. 

[0058] Figure 7 illustrates interference and benign color clusters for a 

foreground text or graphics block candidate region 704 that intersects a background 
image segment 702. Color clusters with segments A, B, C of the image 702 that 
overlap with the text block 704 are interference clusters. Color clusters with segments 
D and E of the image 702 that do not overlap with the text block 704 are benign 
clusters. However, since segment C of the image 702 is associated with the same 
color cluster as the image segment C, the image segment C is treated as an 
interference cluster even though it does not overlap with the text block 704. 

[0059] Continuing at step 207 with the image data clustered into a set of 

interference and benign clusters, a design function (or method) is evaluated to 
recommend a foreground color for the candidate background region, the details and 
embodiments of which are discussed below. The design function (and method) is 
based on one or more predetermined criteria. 

[0060] If an acceptable or a superior candidate background color is identified 

at 208, the system may automatically combine image data with foreground text and/or 
graphics object having the recommended color to produce output a single image or an 
overlaid image (e.g., text and/or graphics overlaying the background image) at 209. If 
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no acceptable or superior result is identified at 208, the user may be presented at 210 
with alternatives to foreground colors to evaluate, and a selection requested at 211. 
The presentation at 210 may include either alternative foreground color 
recommendations and/or examples of the background image overlaid with the 
foreground text and/or graphics object having the recommended color. 

[0061] It is to be understood that the method set forth in Figure 2 need not be 

performed for a single foreground region, but can be used for several foreground 
regions of an image as well. In such a way, it is possible to place more than one text 
or graphics object on a background image or to provide alternative regions with 
corresponding color recommendations for a single text or graphics object. For 
example, Figure 8 illustrates the background image in Figure 3 in which two 
foreground texts, "RED" and "FOX", are provided to the system as foreground text to 
be placed on segmented portions of the background image shown in Figures 5 and 6, 
respectively. In this example, the system determines the recommended color for the 
foreground texts "RED" and "FOX" are red-orange and blue, respectively, which are 
in two different regions of the image. 

[0062] Furthermore, additional properties of the background image can be 

used in order to identify placement of the text or graphics forming part of the 
foreground image on the background. For example, the scale and variance of the 
texture of the background image may play a role in the scale chosen for foreground 
text of a certain size and semantic analysis of the background image might prohibit 
the placing of foreground text or graphics over human faces forming part of the 
background image. 

[0063] With reference again to Figure 1, in one embodiment steps 201, 202 

and 203 set forth in Figure 2 are performed in color clustering module 101, steps 204 
and 205 are performed by image segmentation module 102, and the remaining steps 
set forth in Figure 2 are performed by color selection module 103 and coalescing 
module 118. In this embodiment, the coalescing module 118 may receive as input 
foreground text and/or graphics (e.g., text "RED FOX") and the background image 
111 (e.g., the image shown in Figure 3) and output in addition to or in place of 
recommended color or colors 115, an output image that coalesces the background 
image 1 1 1 and the foreground text and/or graphics (e.g., the image in shown in Figure 
8). 



-13- 



[0064] B. Criteria For Determining Foreground Color 

[0065] Referring again to evaluating a design function at 207 in Figure 2, 

several embodiments for recommending or selecting a color for a candidate region of 
a background image are described below. 

[0066] In a first embodiment for evaluating the design function at 207 

according to one or more predetermined criteria, a legibility criterion is evaluated. In 
this embodiment, the legibility criterion is evaluated by: (A) computing local 
measures of contrast between background and foreground in a neighborhood of a 
candidate region of the background image; and subsequently (B) computing a 
legibility score representative of the lowest few contrast values observed over the 
candidate region. 

[0067] In one embodiment, the lowest few contrast values observed over the 

candidate region is the minimum value of all local contrast measures taken over the 
entire candidate region. In another embodiment, the lowest few contrast values 
observed over the candidate region are those values in the tenth percentile of the 
contrast values taken over the entire candidate region. In yet another embodiment, the 
lowest few contrast values observed over the candidate region is an average those 
lowest contrast values in the lower quartile taken over the entire candidate region. 

[0068] Generally in computing the local measures of contrast, the local 

contrasts are computed over a scale that is proportional to, for example, the 
foreground text size to determine an overall contrast. Subsequently, the overall 
contrast is then multiplied by a factor proportional to the foreground text size. The 
local measures are functions that decrease with increasing local background variation. 
The overall contrast score may be supplemented with additional factors such as the 
intrinsic legibility of a font. 

[0069] Specifically in taking the local measures of contrast, one embodiment 

may involve the following steps: (a) down- sample a region of the image containing 
the text so that uppercase text is approximately thirty pixels high; (b) derive 
normalized versions of the color channels by dividing the red and blue channels by 
two; (c) compute Gaussian weighted averages of the background pixels and the 
foreground pixels over the image, where the Gaussian width is two; (d) compute local 
variances of the Gaussian weighted averages, where variance is computed over a 
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window of size of sixty pixels; (e) estimate the sums of the variances of all three color 
channels; (f) compute a first contrast measure as the squared difference between local 
mean foreground and local mean background colors, divided by the variance estimate; 
(g) compute a second contrast measure by dilating (i.e., taking local maxima of) the 
first contrast measure with a square structuring element (e.g., nine-by-nine pixels); (h) 
compute the output contrast measure by multiplying the second contrast measure by 
the text height. 

[0070] Advantageously, this first embodiment for recommending the 

foreground color using a legibility criterion is adapted to handle multi-color 
background images and foreground text and/or graphics. A further advantage of this 
first embodiment for recommending the foreground color using a legibility criterion is 
that it has been found experimentally to produce legibility comparisons that are in 
good agreement with human perception. 

[0071] Further it will be appreciated that many other uses of this method for 

determining the legibility of an image may be made in addition to recommending 
and/or selecting a foreground color for an image, such as: (a) to proof an existing 
layout for legibility; (b) to select the position of placement of text given a text color 
and the image; (c) to select a size of text of a given text color and the image; (d) to 
select the image from a set of alternative images on which to place text, given the text 
color; (e) to select a "screen" (i.e., region of reduced background contrast) or direction 
in which to desaturate the image (e.g., a sepia tone), given the text color. 

[0072] According to this method for determining the legibility of an image, a 

design function may be defined to use the following legibility criterion to recommend 
the foreground color. The legibility criterion is based on a likelihood ratio for two 
models (a) that the image / contains text T and (b) that the image / just contains 
background B 9 which may be given by: 

score (T) = h log Pr(/|r) . 

Under the assumption of independent pixels, the sum over the parts of the image / 
that contains text T which are foreground according to the text model may be given 
by: 
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score (T) = ^ r(x) , with r(x) = h log 



Pr(/Q) I T) 
Pr(/(x) | B) ' 



where h is a number representative of the size of size T , for instance, the baseline 
height in pixels. Pr(/(x) | T) denotes heuristic or other models of likelihoods that 

the image / contains text T at a given pixel x and Pr(/(x) | B) denotes heuristic 

or other models of likelihoods that the image / contains background B at the given 
pixel x . 

[0073] The metric of for legibility is the difference in score to the so-called 

closest incorrect text V as given by: 

legibility = score (T) - max score (7 71 ) , 

wherein for generating an incorrect text T % 9 it is prescribed that it is sufficient for the 
image of the text to change from foreground to background over a small disc (e.g., 
enough to change an "e" to a "c"). Then, legibility reduces to 

legibility = min max r(x + y) , 
wherein C 2 £ is a small disc (with radius € ). 

[0074] Particularly, a Gaussian model can be taken for each point of the image 

with equal variance for each color coordinate, but with different means F and B for 
foreground and background. Then, each term of the score may be given by: 



The means F and B and the variance a 2 of the background are estimated by taking 
local averages of foreground and background in the composite image (at the same 
scale as the "small disc"). For instance, using a Gaussian windowing function G £ (u) 

on the composite image / with text region T and non-text region Tone can take: 




C 



pixels ueT 



and B c (x) = 



pixels «e T 



HG e (x-u) 



J^G^x-u) 



pixels ue T 



pixels weT 
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[0075] In a second embodiment for evaluating the design function at 207 

according to one or more predetermined criteria, a color harmony criterion is 
evaluated. In this embodiment, a color harmony function is defined for recommending 
a foreground color with respect to one or more interference clusters and/or benign 
clusters. The color harmony function can have a form so as to fulfill a predetermined 
design principle such as: the distance to one of the benign clusters is small (repetition 
of elements); or a color harmony principle such as the color lies along a line in RGB 
space which joins the origin and the center of one of the interference clusters (i.e., 
monotonic color harmony); the color is nearly opposite in hue to a specific color 
cluster (e.g., a major cluster comprising a majority of the pixels), either interference 
or benign (complementary color harmony); the color's hue and the hue of other 
clusters divide the color wheel into equal increments (e.g., triadic and quadric color 
harmony). The color harmony function can take the value 1 if a color adheres to a 
specific color model and 0 if not. For example, if c = kq i where g i <= Q i is the color 
of a benign cluster Q i , then the value of the corresponding color harmony function is 

[0076] In evaluating the design function at 207 according to one or more 

predetermined criteria, other design criteria and/or combinations of such design 
criteria including color harmony and legibility may be used as well which could be 
based on, for example, traditional design principles, physiological phenomena, 
cognitive experiment data, or machine-learned relationships from examples given to 
the system. According to one alternative embodiment, a color harmony criterion and a 
legibility criterion can be applied together to select a foreground color. Ideally in 
these embodiments, the recommended foreground color at 207 should fall into a 
section in color space which is sufficiently far away from all interference clusters 
such that it is legible but, preferably, related to at least one of the interference clusters 
in a way which is harmonious. 

[0077] In a third embodiment for evaluating the design function at 207, a 

combination of color harmony and legibility criteria is used in a design method to 
recommend a color for a candidate region, for which a flow diagram is shown in 
Figure 9. The design method involves initially selecting, at 901, a harmonious color 
set with respect to clustered colors of the background image using a color harmony 
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function, as described in more detail above. Subsequently, the harmonious color set 
(i.e., a subset of the clustered colors) is tested using a legibility function at 902 from 
which one or more colors are selected for recommendation at 903. 

[0078] In an example embodiment combining the evaluation of color harmony 

and legibility criteria, the harmonious color set may be selected (as shown in Figure 9 
at 901) by determining a subspace (set of colors) of the color space where a given 
color model (e.g., monotonic, triadic, etc.) is valid. In this way, a large number of 
colors can already be eliminated. It is even possible to sub-sample this subspace to 
further reduce the number of possible candidate colors. Then at 902 in Figure 9, after 
determining a subspace of the color space, the maximum (or a value sufficiently 
large) of the legibility function is determined for the colors of this subspace. 

[0079] There might not necessarily be a unique maximum of the legibility 

function. One can try to solve this problem by searching for the maximum of the 
legibility function over several color harmony models. Alternatively or additionally 
(if there is still no unique maximum), it may be tested for example at step 208 (shown 
in Figure 2) whether the resulting maximum is actually unique. If yes, the method 
would proceed to step 209 where the background image is output (e.g., displayed, 
printed, transmitted, etc.) together with a foreground text or graphics that has been 
provided to the system. If the maximum is not unique, the possible maxima can be 
presented to a user at step 210. In a following step 211, then, the user can be requested 
to select a color among the suggestions from the system. Otherwise, the maxima can 
be chosen randomly or by another criterion. 

[0080] In a fourth embodiment for evaluating the design function at 207, a 

combination of color harmony and legibility criteria are maximized in a linear design 
function to produce, a recommended color. One possible form of the linear design 
function may be defined as: 

Ya t Kc t P t ) + ± Yk h{c,K k ), 

where P. denote the interference clusters, K k denotes all clusters, both benign and 
interference, / is a legibility function in color space, h is a color harmony function, 
and a i and y k are weighting factors. 
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[0081] The legibility and color harmony functions forming part of the linear 

design function can themselves be combinations of other functions. The color 
harmony and legibility functions can have a form as described above, in particular, 
with predetermined weightings of the coordinates in color space such as a higher 
weighting of the luminance than hue or saturation. 

[0082] C. Miscellaneous 

[0083] The configuration of the described system 100 is intended to be 

modular in nature so that it can be combined with other document quality measures 
and integrated into a larger document layout system. 

[0084] Using the foregoing specification, the invention may be implemented 

as a machine (or system), process (or method), or article of manufacture by using 
standard programming and/or engineering techniques to produce programming 
software, firmware, hardware, or any combination thereof. It will be appreciated by 
those skilled in the art that the flow diagrams described in the specification are meant 
to provide an understanding of different possible embodiments of the invention. As 
such, alternative ordering of the steps, performing one or more steps in parallel, 
and/or performing additional or fewer steps may be done in alternative embodiments 
of the invention. 

[0085] Any resulting program(s), having computer-readable program code, 

may be embodied within one or more computer-usable media such as memory devices 
or transmitting devices, thereby making a computer program product or article of 
manufacture according to the invention. As such, the terms "article of manufacture" 
and "computer program product" as used herein are intended to encompass a 
computer program existent (permanently, temporarily, or transitorily) on any 
computer-usable medium such as on any memory device or in any transmitting 
device. 

[0086] Executing program code directly from one medium, storing program 

code onto a medium, copying the code from one medium to another medium, 
transmitting the code using a transmitting device, or other equivalent acts may involve 
the use of a memory or transmitting device which only embodies program code 
transitorily as a preliminary or final step in making, using, or selling the invention. 

[0087] Memory devices include, but are not limited to, fixed (hard) disk 
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drives, floppy disks (or diskettes), optical disks, magnetic tape, semiconductor 
memories such as RAM, ROM, Proms, etc. Transmitting devices include, but are not 
limited to, the Internet, intranets, electronic bulletin board and message/note 
exchanges, telephone/modem based network communication, hard-wired/cabled 
communication network, cellular communication, radio wave communication, 
satellite communication, and other stationary or mobile network 
systems/communication links. 

[0088] A machine embodying the invention may involve one or more 

processing systems including, but not limited to, CPU, memory/storage devices, 
communication links, communication/transmitting devices, servers, I/O devices, or 
any subcomponents or individual parts of one or more processing systems, including 
software, firmware, hardware, or any combination or subcombination thereof, which 
embody the invention as set forth in the claims. 

[0089] While particular embodiments have been described, alternatives, 

modifications, variations, improvements, and substantial equivalents that are or may 
be presently unforeseen may arise to applicants or others skilled in the art. 
Accordingly, the appended claims as filed and as they may be amended are intended 
to embrace all such alternatives, modifications variations, improvements, and 
substantial equivalents. 
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